Upcoming Event: Oden Institute Seminar

Convergence, Implicit Bias, and Edge of Stability of Gradient Descent for Neural Networks

Rene Vidal,

Abstract

Neural networks trained by gradient descent from random initialization and without explicit regularization often generalize well despite being highly overparameterized. In this talk, I will present recent work aimed at explaining this puzzling phenomenon by addressing three main questions. First, why does gradient descent efficiently find a global minimizer? We examine how data separation, the degree of overparameterization, and the choice of initialization can produce a benign optimization landscape along the training trajectory, enabling linear convergence. Second, why does the solution found by gradient descent generalize well? We show that, for sufficiently separated data, small initialization, and wide networks, the optimization trajectory enters an alignment phase characterized by neural collapse, followed by convergence to a classifier with provable generalization guarantees. Third, why can training with learning rates far larger than those allowed by classical convergence theory lead to faster convergence and better generalization? We relate this edge-of-stability regime to a self-stabilization mechanism and an implicit bias toward flat minima. Together, these results shed light on how optimization, overparameterization, and large learning rates interact to shape the convergence and generalization of gradient descent for neural networks.

Biography

René Vidal is the Penn Integrates Knowledge and Rachleff University Professor of Electrical and Systems Engineering and Radiology at the University of Pennsylvania, where he directs the Center for Innovation in Data Engineering and Science (IDEAS) and serves as Co-Chair of Penn AI. He is also an Amazon Scholar, Affiliated Chief Scientist at NORCE, and former Associate Editor-in-Chief of IEEE Transactions on Pattern Analysis and Machine Intelligence. His research advances the mathematical foundations of deep learning and trustworthy AI, with broad impact across computer vision and biomedical data science. His contributions have been recognized with major honors, including the IEEE Edward J. McCluskey Technical Achievement Award, the D’Alembert Faculty Award, the J.K. Aggarwal Prize, the ONR Young Investigator Award, the NSF CAREER Award, and best paper awards in machine learning, computer vision, signal processing, control, and medical robotics. He is a Fellow of ACM, AIMBE, IEEE, IAPR, and Sloan Foundation.